Systems and methods for identifying users of devices and customizing devices to users

ABSTRACT

A system and method for identifying a user of a device includes comparing audio received by a device with acoustic fingerprint information to identify a user of the device. Image data, video data and other data may also be used in the identification of the user. Once the user is identified, operation of the device may be customized based on the user. Further, once the user is identified, data can be associated with the user, for example, usage data, location data, gender data, age data, dominant hand data of the user, and other data. This data can then be used to further customize the operation of the device to the specific user.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims the benefit ofpriority of, U.S. Non-provisional patent application Ser. No.15/082,104, filed on Mar. 28, 2016, and entitled “Systems and Methodsfor Identifying Users of Devices and Customizing Devices to Users,”which is a continuation of, and claims the benefit of priority of, U.S.Non-provisional patent application Ser. No. 13/838,572, filed on Mar.15, 2013, and entitled “Systems and Methods for Identifying Users ofDevices and Customizing Devices to Users,” which has since issued asU.S. Pat. No. 9,299,350, the contents of each of which is expresslyincorporated herein by reference in its entirety.

BACKGROUND

With the advancement of technology, the use and popularity of electronicdevices, such as smart phones, has increased considerably.Human-computer interactions have even progressed to the point wherehumans can control these devices, and provide input to these devices, byspeaking. Devices are designed to be marketed to the general public andtypically include generic user interfaces and operating instructionsthat appeal to the general masses of users.

BRIEF DESCRIPTION OF DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description taken in conjunction with theaccompanying drawings.

FIG. 1 illustrates a pictorial diagram of a method and system foridentifying a user of a device according to one aspect of the presentdisclosure.

FIG. 2 is a block diagram conceptually illustrating a computing deviceaccording to one aspect of the present disclosure.

FIG. 3 is a block diagram conceptually illustrating a userdetection/identification system according to one aspect of the presentdisclosure.

FIG. 4 illustrates an audio waveform processed according to one aspectof the present disclosure.

FIG. 5 illustrates a computing network for use with distributedprocessing according to one aspect of the present disclosure.

FIG. 6 is a block diagram conceptually illustrating a local device and aremote device for use in a distributed processing environment accordingto one aspect of the present disclosure.

FIG. 7 illustrates a flow diagram of a method and system for identifyinga user of a device according to one aspect of the present disclosure.

FIG. 8 illustrates a flow diagram of a method and system for confirmingan identity of a user of a device according to one aspect of the presentdisclosure.

FIG. 9 illustrates a flow diagram of a method and system for confirmingan identity of a user of a device according to one aspect of the presentdisclosure.

FIG. 10 illustrates a flow diagram of a method and system for confirmingan identity of a user of a device according to one aspect of the presentdisclosure.

FIG. 11 illustrates a flow diagram of a method and system forcustomizing operation of a device of an identified user according to oneaspect of the present disclosure.

FIG. 12 illustrates a flow diagram for identifying voices andassociating location data to the identified voices according to oneaspect of the present disclosure.

FIG. 13 illustrates a flow diagram for customizing operation of a deviceof an identified user using background noise according to one aspect ofthe present disclosure.

DETAILED DESCRIPTION

Typically, computing devices are designed to be marketed to the generalpublic. Thus, devices are configured for use by a general computingpublic with little device customization. While individual devices may becustomized by their users after purchase, such customization may involvesignificant manual input by a user and the resulting customization isgenerally limited to the specifically configured device. Aspects of thepresent disclosure are directed to methods and systems to detect andidentify a user of a device. Once the user is identified, operation ofthe device may then be configured according to the specific user'sprofile, thereby allowing customization of device to the user. Theconfiguration may be performed on local devices through cooperation of aremote device, which may provide the ability to configure a device for avariety of users, even users who may have previously not used the devicebefore. A number of techniques, including audio recognition of the user,may be employed. Other techniques may also be used to verify and/orsupplement the audio recognition. The techniques described herein may beapplied to a variety of computing devices, such as telephone computingdevices, laptop computers, cellular phones, personal digital assistants(PDAs), tablet computers, other mobile devices, server-client computingdevices, mainframe computing devices, etc.

In one aspect of the present disclosure, a user of a device isidentified using voice fingerprinting techniques, and usage datacorresponding to the user is collected and analyzed to customize thebehavior and operation of the device to the specific user. For example,the user interface of the device may be modified to suit the specificuser. In another example, customized data, for example, targetedadvertisements, and other data may be set to the device. In anotherexample, specific user data such as contacts, calendar information, orother user specific information may be accessed by the device whenoperated by the specific user. The user specific information may bestored on a local device or on a remote device. If stored on a remotedevice, the user specific information may be accessed by the remotedevice, may be downloaded by the local device when being operated by thespecific user, or some combination thereof. In another aspect, thedevice may adjust its input and/or output processes based on theidentity of a user. For example, if the user's profile indicates thatshe is left-handed and typically uses light strokes to interact with atouch screen of the device, once the user is identified, the device mayadjust the sensitivity of its touch screen to account for the user'stouch characteristics. In another example, a device may adjust a screenoutput font size based on a user's history or preference of item viewingsize as indicated in the user's profile.

FIG. 1 illustrates an exemplary implementation of the system and method.A device 102, for example, a mobile device, may include one or moreinput devices, such as a microphone, an inertial sensor, a videorecorder, a camera, and other input devices that may be used to identifythe user of the device. In one aspect, the device 102 may receive aspoken audio input from a user 104 via a microphone of the device 102.The audio input may be compared to a profile of one or more users basedon previous audio inputs of the user(s). Each user's profile may includea characterization of the user's speaking input that may be used toidentify the user among other users. This characterization may bereferred to as a voice signature, a voice fingerprint, or an acousticfingerprint. Audio input into the device may be compared to voicefingerprint data of users, as shown in block 106, to identify thecurrent user of the device, as shown in block 110. In some aspects, thedevice 102 may also receive secondary data (for example, image data,video data and/or usage/touch characteristics, etc.) of the user 104through a camera, a video recorder and/or inertial sensors of thedevice. The device 102 may then compare the secondary data to availableimage, video and/or usage/touch characteristic data of user(s), as shownin block 108, as a supplement to the audio data to identify the currentuser, as shown in block 110. For example, the device 102 may compare theimage data with a stored image of the user 104 to determine that theuser 104 is the one speaking; the device 102 may analyze the video dataand compare lip and mouth movements of the user to determine that theaudio data received by a device's microphone is actually coming from thedevice's user 104 rather than someone also near the microphone; and/orthe device 102 may compare the usage/touch characteristics with a storeduser usage/touch profile of the user 104 to determine that the user 104is the using the device 102. Other information, such as data from aninertial device, locator device, touchscreen, etc. may also be used toconfirm the identity of the user.

Once the user is identified, the operation of the device may becustomized to the user, as shown in block 112. In addition, once acurrent user has been identified, the operation of the device user maybe associated with the user and analyzed so that future customization ofthat device (or other devices) may take into account the user'soperation of the device, thereby enabling the device to be moreparticularly customized to suit the needs and tendencies of the user.

The operations of FIG. 1 may be performed locally on a device beingoperated by a user, or may be performed in whole or in part on a remotedevice which is connected to the device being operated by a user. Forexample, a remote device may store the acoustic fingerprint data used toidentify users. A more detailed description of a distributed environmentusing a local device and one or more remote devices is discussed below.

Aspects of the present disclosure may allow for customization of theoperation of devices to suit the particular needs of the specific users.This may improve the efficiency of the devices as well as the efficiencyof the users themselves. In one example, the acoustic fingerprint of theuser may be stored in a user profile for the user, and used to identifythe user of the device based on the user's voice. Upon identifying thecurrent user of the device, the user interface of the device may becustomized for operation based on a number of user specificcharacteristics described below. The operation of the device may also becustomized in numerous other ways as also described in further detailbelow.

Aspects of the present disclosure may be implemented as a computerimplemented method in a computing device or computer system. FIG. 2illustrates an exemplary computing device 200. The computing device 200may include a processor 202 in communication with a variety of othercomponents over a system bus 204 or through a direct connection. Theother components may include, for example, a network interface 206, oneor more input and/or output devices 208, a display interface 210, and astorage database 212. As appreciated by those skilled in the art, thenetwork interface 206 enables the computing device 200 to communicatedata, control signals, data requests, and other information with otherresources including computers, data sources, storage devices, and thelike, on a computer network such as the Internet. The network interface206 may be configured to communicate via wired or wireless connections.As one skilled in the art should appreciate, the computing device 200may receive an audio, image, video, and other inputs and transmit theinput data to another computer, a storage device, or other source viathe computer network, as well as save the input data to a networkedlocation.

The input and/or output devices 208 enable the computing device 200 toobtain data input from a variety of devices including, but not limitedto, a digital pen, a touch screen, a keyboard, a mouse, a scanner, amicrophone, an inertial sensor, a video recorder, a camera and the like.In addition to the exemplary components described above, a displayinterface 210 may be used for outputting display information to a user.Typically, the display information is output by the display interface210 via a display device (e.g., a screen or similar device). While notillustrated, one skilled in the art should appreciate that a displaydevice may be incorporated as an integral element within a computingdevice 200 or may be separate therefrom.

The processor 202 may be configured to operate in accordance withprogramming instructions stored in a memory 214. The memory 214generally comprises RAM, ROM, and/or other memory. Thus, in addition tostorage in read/write memory (RAM), programming instructions may also beembodied in read-only format, such as those found in ROM or otherpermanent memory. The memory 214 may store an operating system 216 forcontrolling the operation of the computing device 200. The operatingsystem may be a general purpose operating system such as a MicrosoftWindows operating system, a UNIX operating system, a Linux operatingsystem, or an operating system specifically written for and tailored tothe computing device 200. Similarly, the memory 214 may also storeuser-executable applications 218, or programs, for conducting variousfunctions on the computing device 200. For example, the application 218in memory 214 may be configured according to aspects of the presentdisclosure to identify a user of the device and customize the operationof the device to the user.

A user detection/identification system may be implemented in anapplication or module implemented on the computing device 200, in aplurality of cooperating applications/modules on the computing device200, or in a plurality of cooperating applications and/or modulesdistributed over multiple computing devices 200 in a computer network,for example. Components of the user detection/identification system maybe identified with regard to various logical components. The device 200may also include a user detection/identification module 220 includingprocessing module(s) 222 and user profile storage 224. The userdetection/identification module 220 may include its own processor and/ormemory or may make use of processor 202 and/or memory 214. The userprofile storage may be separate from storage 212 or may overlap with/beincluded in storage 212.

FIG. 3 is a block diagram illustrating logical components of a userdetection/identification module 220 for detecting/identifying a user ofa device. As illustrated in FIG. 3, the user detection/identificationmodule 220 includes one or more processing module(s) 222 which mayinclude acoustic fingerprint modules 302, one or more usage/toughprocessing modules 304, one or more image processing modules 306, one ormore video processing modules 308 and one or more other processingmodules 310. Each module may include its own processor and/or memory ormay make use of processor 202 and/or memory 214. Each module 302, 304,306 and 308 may be connected to an associated storage component. Forexample, the acoustic fingerprint module(s) may be connected to acousticfingerprint store 312 which stores data identifying users by theiracoustic fingerprints. Usage/tough processing module(s) 304 may beconnected to usage/touch store 314 which stores data identifying usersby their characteristics of touching and using the device. Imageprocessing module(s) 306 may be connected to image store 316 whichstores data identifying users by their images. Video processingmodule(s) 308 may be connected to video store 318 which may store videoassociated with particular users and/or may store video associated withparticular audio segments (such as matching a speaker's words to themovement of his/her mouth) so that input audio may be compared withvideo of a device user to confirm whether received audio is being spokenby a device user. Other processing module(s) 310 may be connected toother store 320 which may store additional data associated with aparticular users so that other data may be compared to confirm whetherreceived audio is being spoken by a device user.

The individual storage components 312, 314, 316, 318 and 320 may beseparate components or may be grouped in some manner with user profilestorage 224 and/or storage 212. In another aspect, other modules and/orstorage components may also be included in the userdetection/identification module 220. Such modules may include componentsfor considering other information when identifying a user of a device.Such information may include location information, device usageinformation, or other kinds of information such as those describedbelow.

The modules 302, 304, 306, 308 and 310 may perform user detection andidentification, but may also perform user profiling/fingerprinting,where the data used to populate stores 212, 312, 314, 316, 318 and 320is generated. For example, the acoustic fingerprint module(s) 302 maytake input audio information to generate an acoustic fingerprint for auser and store it in store 312. Additionally, the acoustic fingerprintmodule(s) 302 may take input audio information to refine an existingacoustic fingerprint for a user stored in store 312. Similar usage/touchcharacteristic data may be populated into usage/touch store 314 byusage/touch processing module(s) 304, image data may be populated intoimage store 316 by image processing module(s) 306, video data may bepopulated into video store 318 by video processing module(s) 308 andother data may be populated into other store 320 by other processingmodule(s) 310.

Once the stores are populated, the user detection/identification modulemay perform user identification using modules 302, 304, 306, 308 and310. For example, the user detection/identification module 220 mayreceive an audio input, such as a spoken input from a user. The inputmay be received through a microphone of the device. The acousticfingerprint module 302 may analyze the audio input of the spoken inputand generate a signature or voice/acoustic fingerprint of the voice ofthe user that spoke the input. The acoustic fingerprint may be generatedusing any of numerous acoustic fingerprinting techniques. In oneexample, the audio input is analyzed and a digital summary of attributesof the audio is created, for example, using frequency, intensity andother attributes of the audio input. FIG. 4, illustrates a digitizedaudio waveform 402, with multiple points 406 of a first portion 404 asthe audio waveform 402 is being processed. The audio qualities of thosepoints, for example, frequency and intensity, as well as the user'stypical voice interactions with a device may be used in generating thevoice/acoustic fingerprint of the user. The audio qualities of thepoints may also be combined into a matrix that represents a time periodor length of the audio. A number of approaches known to those skilled inthe art may be used to process the audio data.

The usage/touch processing module 304 may process usage/touchcharacteristics of the current user of the device, for example,collected with an inertial sensor of the device to that can be used tomeasure and/or calculate and record the amount of pressure the usertouches the screen with, the acceleration, the velocity and theorientation of the device. Usage/touch characteristics includecharacteristics which describe how a user typically interacts with thetouch screen of a device or how a user physically interacts with thedevice. Such characteristics may include if a user typically users aright or left hand to interact with the device, how much pressure theuser typically applies when interacting with the device, the speed ofthe user's touch strokes on the device, the thickness of the user'sstrokes on the device (which may correlate to both the pressure of theuser's touch as well as the size of his/her fingers), the angles atwhich the user approaches a touch screen of the device, the grip of theuser on the device, how steady the user typically holds the device, etc.Such usage/touch characteristics may be compiled to create a set of datawhich may assist in identifying the user or identifying certainqualities of a user (such as gender, dominant hand tendencies, etc.) Theusage/touch characteristic data may be processed to identify the currentuser of the device. In this aspect, one or more usage/touchcharacteristic models for specific users may be created based onusage/touch characteristics of the user stored in that user's profile.In this aspect, the usage/touch characteristics of the current user maybe compared to identified usage/touch characteristics stored in theusage/touch store 314 to identify the current user.

The image processing module 306 may process an image of the current userof the device, for example, taken with a camera of the device. The imagemay be processed using facial recognition, or other known techniques toidentify the current user of the device. In this aspect, one or morefacial recognition models for specific users may be created based on animage of the user stored in that user's profile. In this aspect, theimage of the current user may be compared to identified images of usersstored in the image store 316 to identify the current user.

The video processing module 308 may process video of the current user ofthe device, for example, taken with a video recorder of the device. Inone aspect, the video may be processed to detect mouth and lip movementsof the current user speaking an input into the device. These mouth andlip movements, as well as other video features, may be analyzed andcompared to an audio input to determine whether the audio corresponds tothe spoken input of the user. In this aspect, the video may be used toconfirm that the audio input is in fact coming from the current userrather than audio coming from a different individual. In another aspect,the video may also be processed using facial recognition techniques toidentify the current user of the device. Video data for this processingmay be stored in video store 318.

The usage/touch processing, the video processing and/or the imageprocessing techniques may be combined with the acoustic fingerprintingand audio/voice recognition techniques to identify the current user ofthe device. For example, the facial recognition techniques may provideadditional assurances that the user being identified using the acousticfingerprinting and audio/voice recognition techniques is correct.

In an aspect, the user detection/identification module 220 may receiveinput data and may initiate instructions to othercomponents/applications of a device in order to perform actions, such asperforming user recognition processing or performing configuring ofanother device based on user recognition results. For example, theacoustic fingerprinting, voice recognition, usage/touch processing,image processing and/or video processing may be performed by a localdevice or by a remote device. If performed by a remote device, theremote device may then send instructions to a local device to performoperations.

As shown in FIG. 5, multiple devices may be connected over a network502. Network 502 may include a local or private network or may include awide network such as the internet. Devices may be connected to thenetwork 502 through either wired or wireless connections. For example, awireless device 504 may be connected to the network 502 through awireless service provider. Other devices, such as computer 510, mayconnect to the network 402 through a wired connection. Other devices,such as laptop 506 or tablet computer 508 may be capable of connectionto the network 502 using various connection methods including through awireless service provider, over a WiFi connection, or the like.Networked devices may receive spoken audio through a number of audioinput devices. Audio input devices may be connected to networked deviceseither through a wired or wireless connection. Networked devices mayalso include embedded audio input devices.

In certain system configurations, one device may capture or receive anaudio signal, video signal and/or image signal and other device(s) mayperform the acoustic fingerprinting, voice recognition, usage/touchcharacteristic processing, image processing and/or video processing. Forexample, audio input may be received by computer 510 and sent over thenetwork 502 to computer 512 or server 514 for processing. Or computer510 may partially process the audio signal before sending it over thenetwork 502. In another aspect, the acoustic fingerprinting, voicerecognition, usage/touch characteristic processing, image processingand/or video processing may all be performed on different devices.Because such processing may involve significant computational resources,in terms of both storage and processing power, such split configurationsmay be employed where the device capturing or receiving the input dataor signal has lower processing capabilities than a remote device.

One configuration of the present disclosure is shown in FIG. 6. In thisconfiguration, a first device or local device 600 may include an audiocapture device 602, a usage/touch capture device 604, an image capturedevice 606, a video capture device 608 The audio capture device 602, theusage/touch capture device 604, the image capture device 606 and/or thevideo capture device 608 may be internal to the local device 600 or maybe located external to the local device 600. A second device or remotedevice 610 is in communication with the local device 600 over a network612. The two devices may be examples of computing device 200, or may beother devices. The network 612 may be a wide area network such as theInternet, a local area network, a network allowing physical connectionof the devices, a wireless communication network, or other suitableconnection between the devices including a direct (wired or wireless)connection.

Audio, usage/touch, video, and/or image data is captured by the audiocapture device 602, the usage/touch capture device 604, the imagecapture device 606 and/or the video capture device 608 and an audiosignal, usage/touch data signal, image signal, and/or video signal isproduced. The signal(s) may be processed by the local device 600 or maybe transmitted over the network 612 to the remote device 610 forprocessing. For example, the local device 600 may capture audio andtransmit the audio over the network 612 to the remote device 610. Theremote device may include user detection/identification module 220 whichperforms user identification as described herein. In another aspect thelocal device may perform some processing on the captured audio,usage/touch, image and/or video data prior to sending to the remotedevice. For example, instead of sending raw data to the remote device610 the local device 600 may send a limited representation of the rawdata, where the representation is sufficient for the remote device 610to perform user identification.

In another aspect, usage data may be collected and transmitted to theremote device 610 for processing. In one aspect, the remote device 610may associate the usage data to the identified user, analyze the usagedata and generate configuration instructions that correspond to acustomized user interface based on the usage data. The configurationinstructions may then be transmitted to and executed by the local device600 to customize the operation of the local device 600 to the currentuser.

In another aspect, the remote device 610 may transmit other userprofiles and including acoustic fingerprints of other users to the localdevice 600. In this aspect, the local device 600 can identify otherusers that may be using the local device 600 and customize the operationof the local device 600 to those specific other users.

FIG. 7 is a flow diagram illustrating an exemplary method for generatingan acoustic fingerprint for a user and storing that fingerprint in auser profile. One or more of the logical components of FIGS. 2 and/or 3may perform the method of FIG. 7. In block 702, the device receives orcaptures audio, for example, via a microphone of the device. The audiois processed to identify voices within the audio, as illustrated inblock 704. Upon determining there is a voice contained in the audio, anacoustic fingerprint is generated based on the voice, as illustrated inblock 706. The acoustic fingerprint may be associated with the specificuser, as illustrated in block 708, and the acoustic fingerprint can bestored in a user profile corresponding to the specific user speaking theaudio, as illustrated in block 710. In an aspect, a device may include asetting option, that allows a user to specifically record the user'svoice in a controlled environment. This may allow the device to receivea higher quality audio signal and generate a higher quality acousticfingerprint for the specific user. The recorded voice may be used togenerate an audio fingerprint to be stored in the corresponding userprofile. Once user profiles are generated they may be compared toincoming audio from users to identify the user of a device.

In one aspect, once input audio is analyzed as part of a useridentification process, it may be desirable to further analyze the audioand/or one or more additional inputs which may be used to confirm theuser identification from the audio input. For example, a device may wishto discard portions of input audio that include speech from a sourcewhich not actually the user of a device (such as an individual standingnear the device, a song or video being played by the device, etc.). Inone aspect, a device may correlate audio input with a known audio beingoutput by or near a device (such as a music app playing while a deviceis receiving speaking input). If there is a strong correlation betweenthe audio input and the known audio being output, the audio input may bediscarded.

In another aspect, secondary data, for example, image data, video data,user usage/touch data, user usage history data, location data, and otherdata can be collected and compared to stored data in a user profile. Forexample, an image of the current user of a device may be taken by acamera of the device and compared to stored images of users. FIG. 8 is aflow diagram illustrating an exemplary method for confirming a user ofthe device using secondary data of the user. In block 802, the devicereceives or captures audio, for example, via a microphone of the device.The received audio signal may then be compared to available acousticfingerprint(s) of user(s) to preliminarily identify a user, illustratedas block 804. The device may also take or receive secondary data (e.g.,image data, for example, via a camera of the device) corresponding tothe user (e.g., an image of the user), illustrated as block 806. Thesecondary data may be compared to stored data of the user (e.g., astored image of the user), illustrated as block 808. The secondary datamay be used to confirm that the audio is coming from the same user(e.g., as shown in the image), illustrated as block 810. For example, ifthe secondary data does not match the stored data of the user, the audioprobably may not correspond to the user, illustrated as block 812. Whenthe secondary data matches the stored data of the user, the audioprobably likely corresponds to the user. When the audio does correspondto the user, the identity of the specific user may be confirmed,illustrated as block 814, and the device may be configured for thespecific user, illustrated as 816.

In another aspect, a video of the current user may be taken by a videorecorder of the device. FIG. 9 is a flow diagram illustrating anexemplary method for confirming a user of the device using a videorecording. In block 902, the device receives or captures audio, forexample, via a microphone of the device. The received audio signal maythen be compared to available acoustic fingerprint(s) of user(s) topreliminarily identify a user, illustrated as block 904. The device canalso take or receive video data, for example, via a video recorder ofthe device, corresponding to a video of the user, illustrated as block906. The video data can be analyzed to detect mouth and lip movements ofthe user while speaking an input contained in the audio, illustrated asblock 908. The video data can be used to compare the audio to the mouthand lip movements to confirm that the audio corresponds to the spokeninput of the user, illustrated as block 910. For example, if the videodata does not correspond to the spoken input of the user, the audioprobably does not correspond to the user, illustrated as block 912. Whenthe video data corresponds to the spoken input of the user, the audioprobably does correspond to the user. When the audio does correspond tothe user, the identity of the specific user may be confirmed,illustrated as block 914, and the device may be configured for thespecific user, illustrated as 916.

In an aspect, the local device may include additional input devices, forexample, touch sensors and inertial sensors that may be used to recorduser specific data (e.g. user usage/touch characteristics) such as theamount of pressure the user touches the screen with, acceleration,velocity and/or the orientation of a device when used by a particularuser. The device may also include a location component which determinesthe location of a device while being used by a user. Other suchcomponents and information may be gathered. This data may be used toboth populate a user profile and to compare input data to existing userprofiles to identify a user. For example, these inputs may also beprocessed by the device or be transmitted to a remote device forprocessing of the morphology of the user. This morphology may relate totouch characteristics of the user, for example, the strength or pressureof the touch, the length of touch (in terms of both time of contact witha touch screen and length of typical strokes on the touch screen), thesmoothness of the touch (such as how a user's touch on the device variesor stays consistent during a particular touch stroke), the preferencefor the left or right hand, or the like. This morphology may be acombination of aspects of the user, such as, the size, weight, strength,and gender of the user, and can be factored into the customization ofthe device. For example, vibration of the device may be observed by theinertial sensor when the user touches and taps on the device. A sharpvibration or acceleration of the device may correspond to a women, forexample, and a smoother longer vibration of the device may correspond toa larger male, for example. By analyzing the data collected by the touchsensors and inertial sensors, the analyzed data can be factored into theidentification of the user as well as factored into the customization ofthe device. This may allow the device to be adapted more precisely tothe morphology of the user.

In an aspect, user usage/touch characteristics of the current user maybe collected and recorded by a touch sensor and/or inertial sensor ofthe device. FIG. 10 is a flow diagram illustrating an exemplary methodfor confirming a user of the device using user usage/touchcharacteristics. In block 1002, the device receives or captures audio,for example, via a microphone of the device. The received audio signalmay then be compared to available acoustic fingerprint(s) of user(s) topreliminarily identify a user, illustrated as block 1004. The device canalso take or receive user usage/touch characteristic data, for example,via a touch sensor and/or inertial sensor of the device, illustrated asblock 1006. This user usage/touch characteristic data may correspond toan amount of pressure the user touches the screen with, the length andthe smoothness of the touch, acceleration, grip, velocity and/or theorientation of the device when touched by the user, and other datacorresponding to the usage/touch characteristics of the user.

The user usage/touch characteristic data can be analyzed and compared toa stored user usage/touch profile associated with the specific user,illustrated as block 1008. The user usage/touch characteristic data canbe used to confirm that the audio corresponds to the specific user,illustrated as block 1010. For example, if the user usage/touchcharacteristic data does not correspond to the user, the audio probablydoes not correspond to the user, illustrated as block 1012. When theuser usage/touch characteristic data corresponds to the user, the audioprobably does correspond to the user. When the audio does correspond tothe user, the identity of the specific user may be confirmed,illustrated as block 1014, and the device may be configured for thespecific user, illustrated as 1016.

In an aspect, both positive and negative confirmations/associations(e.g., when the audio corresponds or does not correspond to the specificuser) of the user may be used as feedback inputs to increase theaccuracy of the acoustic fingerprint and the user profile. Thus, theacoustic fingerprint and user profile can be continuously updated andtailored to the specific user, which may then feed into theidentification of the user making the identification more accurate andefficient.

As illustrated above, following identification of a user of a device,the device may then be customized for use by the specific user. Forexample, a user's personalized information (such as identity, phonenumber, etc.) may be associated with the device while the device isbeing used by the user. Further, applications that are frequently usedby the user may be activated for the device. Other information such ascontacts, calendar entries, graphical user interface preferences,operational preferences, wireless carrier information, etc. may beactivated/made available to the device. In one aspect this data may besent to the user operated device from a remote device. In another aspectthis data may be resident on the local device operated by the user, butonly activated once the user is recognized using the techniquesdescribed herein. In another aspect some data may be sent to the deviceand other data may be resident on the device and activated upon useridentification. User configuration data may be stored as part of userprofile storage 224, in storage 212 or as part of one or more otherstorage component(s).

When a particular user has been associated with a device, the operationof the device by the user may be used to populate a user profileassociated with the user that may then be used for further customizationof the device based on the preferred operation of the device by theuser. FIG. 11 is a flow diagram illustrating an exemplary method foridentifying a user and then customizing the device to the current userbased on the user's usage data. In block 1102, the device receives orcaptures audio, such as, a voice command, via a microphone of thedevice. The user of the device is then identified using acousticfingerprinting, illustrated as block 1104. The acoustic fingerprintingmay include comparing the voice input to one or more acousticfingerprints of one or more users of the device, which may be stored ina user profile corresponding to the individual users. The identity ofthe current user is then identified by determining which acousticfingerprint matches the voice input. Further, any of the methodsdescribed above for confirming that the voice command is from the usercan be used, including taking an image of the current user with a cameraof the device and recording a video of the current user issuing thevoice command, etc.

Usage data corresponding to the use of the device may be collected andassociated with the current user, illustrated as block 1106. The usagedata may also be stored in the user profile corresponding to the currentuser, illustrated as block 1108. This usage data may be analyzed tocustomize the operation of the device to the current user, illustratedas block 1110. For example, the usage data may indicate whichapplications the user frequently utilizes (e.g. email, video, internetsearching, etc.), how often the user uses the device, and what the useruses the device for. Based on this information the behavior andoperation of the device may be customized to the specific preferencesand uses of the current user. The usage data may be collected prior toidentifying the user and associated with the user upon identification ofthe user and/or collected after identifying the user and associated withthe user.

In an aspect, the user profile may include other information about theuser that may be aggregated with the usage data to further refine thebehavior and operation of the device. For example, the user profile mayinclude such information as, gender, age, height, weight, dominant hand(e.g. left handed, right handed, ambidextrous), personal interests,hobbies, and other information about the user. Any part of or all of theinformation about the user can be factored into the customization of thedevice to enhance the user experience.

In an aspect, location data may be collected and incorporated into thecustomization of the device. FIG. 12 is a flow diagram illustrating anexemplary method for customizing the device to a current user usinglocation data. In block 1202, the device receives or captures audio,such as, a voice command, via a microphone of the device. The user ofthe device is then identified using acoustic fingerprinting, illustratedas block 1204. Location data corresponding to a geographic location ofthe device may be collected, for example, using a Global PositioningSystem (GPS), illustrated as block 1206. This location data may be usedto customize the operation of the device to the current user,illustrated as block 1208. For example, the location data may be used topresent dining options, traffic information, and other information tothe user based on the user's location. The location data may also beused to identify the types of stores and places the user frequentlyvisits, which can be used to infer the likes and dislikes of the user.Like the usage data, the location data may be collected prior toidentifying the user and associated with the user upon identification ofthe user and/or collected after identifying the user and associated withthe user.

In an aspect, audio input into a device may include multiple voices. Inthis aspect, voices contained in the background may be recognized andused to customize the device to the user based on the environment theuser is in. FIG. 13 is a flow diagram illustrating an exemplary methodfor recognizing background or additional voices. In block 1302, thedevice receives or captures audio via a microphone of the device. Theaudio is processed for voices, illustrated as block 1304. A current userof the device is identified based on at least one of the voices in theaudio matching the voice of the user to an acoustic fingerprint,illustrated as block 1306. Additional voices in the audio are detected,illustrated as 1308. The additional voices may be associated to voicesof children, for example, the children of the user, illustrated as block1310. The additional voices may be associated to voices of friends orcolleagues of the user, illustrated as block 1312. The additional voicesmay be associated to a location of the user, illustrated as block 1314.For example, the location of the user may be determined based on thevoices detected. If the voices are voices of the user's children, theuser is likely not at work, and may be at home. Similarly, if the voicesare voices of the user's work colleagues, the user is likely at work,etc. The data associated with the child voices, friend and colleaguevoices, and location data may be stored in a user profile correspondingto the current user, illustrated as block 1316.

In an aspect, the additional or background voices that are detected maybe correlated to certain individuals based on acoustic fingerprints thathave been generated for those individuals. This can be used by thedevice to identify not only the current user, but also identify thepeople around the user. In another aspect, the detection of additionalor background voices may be used in the identification of the currentuser. For example, based on the voices in the background, the currentuser of the device may be inferred. Alternatively, the background voicescan be used to infer who the current user is not. Thus, even withoutusing the acoustic fingerprinting, the current user can be identified.

In an aspect, the detection of additional or background voices may beused in the customization of the behavior and operation of the device.For example, if the current user is watching a movie not suitable forchildren (e.g., a R-rated movie) and the device collects backgroundvoices of children, the device may be customized to automatically stopplaying the movie. Similarly, if the current user is looking up a movieto watch, and the device collects background voices of children, thedevice may be customized to automatically remove movies that are rated Rfrom the available movie listing.

In an aspect, the detection of additional or background voices may beused in the customization of the behavior and operation of the device.For example, the device may access and/or receive acoustic fingerprintsand user profiles of others. This allows the device to identify otherusers. In this aspect, a device of a first user may identify a seconduser and may be customized to the second user when the second user isusing the device. This allows the device to be customized to friends,family members, etc. of the user when they happened to be using thedevice.

In an aspect, the device may be configured to detect other devices thatare near the current user. In this aspect, the device may be able toidentify common areas of interest between the users of the devices andmake suggestions for videos, music, games, events, and otherentertainment that are common or would be of interest to the varioususers of the devices.

In an aspect, a single device may include multiple users, for example, afirst user, a second user, etc., and thus multiple user profiles may beassociated with the device. Each user profile may include an acousticfingerprint corresponding to the specific user. This allows the deviceto distinguish and identify who the current user is and presentcustomized behavior and operation tailored to the specific current user.The current user may then switch among a first user and other users, andthe customization of the device may also switch accordingly. In anotheraspect, the device may detect multiple current users of the device atsame time. In this aspect, the device may be customized based on anaggregation of the multiple users' profiles to enhance the experience ofthe users' as a whole. For example, the device may suggest videos,songs, and games to play that are of common interest to the multipleusers.

In an aspect, user identification may assist with security features of adevice. For example, if User A is the only authorized user of thedevice, when a User B attempts to use the device, the device will beable to detect that the User A is not the current user of the device.When the device detects that User A is not issuing the audio input, thedevice may disallow the input, lock the device, and or otherwise preventthe User B from utilizing the device.

In another aspect, the identification of the user may be useful indetermining which choice commands to execute. For example, if thecurrent user is in a crowded place with other people that are usingother devices, the device may be receiving numerous audio commands thatare being over heard from other users. In this aspect, the voice commandfrom the current user can be identified and executed without the deviceaccidentally executing an incorrect command being spoken by from someonenear the current user.

The above aspects of the present disclosure are meant to beillustrative. They were chosen to explain the principles and applicationof the disclosure and are not intended to be exhaustive or to limit thedisclosure. Many modifications and variations of the disclosed aspectsmay be apparent to those of skill in the art. Persons having ordinaryskill in the field of computers, digital imaging, and/or contentconversion, should recognize that components and process steps describedherein may be interchangeable with other components or steps, orcombinations of components or steps, and still achieve the benefits andadvantages of the present disclosure. Moreover, it should be apparent toone skilled in the art, that the disclosure may be practiced withoutsome or all of the specific details and steps disclosed herein.

Aspects of the present disclosure may be implemented as a computerimplemented method, a system, or as an article of manufacture such as amemory device or non-transitory computer readable storage medium. Thecomputer readable storage medium may be readable by a computer and maycomprise instructions for causing a computer or other device to performprocesses described in the present disclosure. The computer readablestorage medium may be implemented by a volatile computer memory,non-volatile computer memory, hard drive, solid state memory, flashdrive, removable disk, and/or other media.

Aspects of the present disclosure may be performed in different forms ofsoftware, firmware, and/or hardware. Further, the teachings of thedisclosure may be performed by an application specific integratedcircuit (ASIC), field programmable gate array (FPGA), or othercomponent, for example.

Aspects of the present disclosure may be performed on a single device ormay be performed on multiple devices. For example, program modulesincluding one or more components described herein may be located indifferent devices and may each perform one or more aspects of thepresent disclosure. As used in this disclosure, the term “a” or “one”may include one or more items unless specifically stated otherwise.Further, the phrase “based on” is intended to mean “based at least inpart on” unless specifically stated otherwise.

What is claimed is:
 1. A computer-implemented method, comprising:receiving first audio data corresponding to audio in an environment of afirst device; receiving video data associated with the first audio data;processing the video data to determine mouth movements of an individual;determining that a first portion of the first audio data corresponds tothe mouth movements; determining that the first portion of the firstaudio data corresponds to previously stored voice data associated with aprofile of a user; and determining that the individual is the user basedat least in part on the first portion of the first audio datacorresponding to the mouth movements of the individual and thepreviously stored voice data.
 2. The computer-implemented method ofclaim 1, further comprising: based at least in part on determining thatthe individual is the user, configuring operation of the first devicebased at least in part on additional data associated with the profile.3. The computer-implemented method of claim 2, wherein the additionaldata comprises user-specific information associated with the profile,and configuring the operation of the first device further comprisespermitting the first device to access the user-specific information. 4.The computer-implemented method of claim 3, wherein the user-specificinformation is stored on a remote device, and configuring operation ofthe first device further comprises transferring the user-specificinformation from the remote device to the first device.
 5. Thecomputer-implemented method of claim 2, wherein the additionalinformation comprises at least one device setting of the first device,and configuring the operation of the first device further comprisescausing the first device to operate based at least in part on the atleast one device setting.
 6. The computer-implemented method of claim 5,wherein the at least one device setting comprises a touchscreensensitivity of a touchscreen of the first device.
 7. Thecomputer-implemented method of claim 5, wherein the at least one devicesetting comprises a font size for text displayed on a screen of thefirst device.
 8. The computer-implemented method of claim 1, furthercomprising: receiving image data representing an image acquired by acamera of the first device; and determining that the image datacorresponds to stored image data associated with the profile, whereindetermining that the individual is the user is further based at least inpart on the image data corresponding to the stored image data.
 8. Thecomputer-implemented method of claim 1, further comprising: receivingfirst device usage data representing a first interaction with the firstdevice; and determining that the first device usage data corresponds tostored usage characteristic data associated with the profile, whereindetermining that the individual is the user is further based at least inpart on the first device usage data corresponding to the usagecharacteristic data.
 9. The computer-implemented method of claim 8,further comprising: prior to receiving the first audio data, receivingsecond audio data corresponding to audio in the environment; determiningthat the second audio data corresponds to the previously stored voicedata; receiving second device usage data representing a secondinteraction with the first device; determining, based at least in parton second audio data corresponding to the previously stored voice data,that the user performed the second interaction; determining, based atleast in part on the second device usage data, the usage characteristicdata, the usage characteristic data representing how the user interactswith the first device; and associating the usage characteristic datawith the profile.
 10. The computer-implemented method of claim 1,further comprising: receiving device usage data representing aninteraction with the first device; determining, based at least in parton determining that the individual is the user, that the user performedthe interaction; determining, based at least in part on the device usagedata, usage characteristic data representing how the user interacts withthe first device; associating the usage characteristic data with theprofile; and configuring operation of the first device based at least inpart on the usage characteristic data.
 11. A computing system,comprising: at least one processor; and at least one computer-readablemedium encoded with instructions that, when executed by the at least oneprocessor, cause the computing system to: receive first audio datacorresponding to audio in an environment of a first device, receivevideo data associated with the first audio data, process the video datato determine mouth movements of an individual, determine that a firstportion of the first audio data corresponds to the mouth movements,determine that the first portion of the first audio data corresponds topreviously stored voice data associated with a profile of a user, anddetermine that the individual is the user based at least in part on thefirst portion of the first audio data corresponding to the mouthmovements of the individual and the previously stored voice data. 12.The computing system of claim 11, wherein the at least onecomputer-readable medium is encoded with additional instructions that,when executed by the at least one processor, further cause the computingsystem to: based at least in part on determining that the individual isthe user, configure operation of the first device based at least in parton additional data associated with the profile.
 13. The computing systemof claim 12, wherein the additional data comprises user-specificinformation associated with the profile, and the at least onecomputer-readable medium is encoded with additional instructions that,when executed by the at least one processor, further cause the computingsystem to: configure the operation of the first device at least bypermitting the first device to access the user-specific information. 14.The computing system of claim 13, wherein the user-specific informationis stored on a remote device, and the at least one computer-readablemedium is encoded with additional instructions that, when executed bythe at least one processor, further cause the computing system to:configure the operation of the first device at least by transferring theuser-specific information from the remote device to the first device.15. The computing system of claim 12, wherein the additional informationcomprises at least one device setting of the first device, and the atleast one computer-readable medium is encoded with additionalinstructions that, when executed by the at least one processor, furthercause the computing system to: configure the operation of the firstdevice at least by causing the first device to operate based at least inpart on the at least one device setting.
 16. The computing system ofclaim 15, wherein the at least one device setting comprises atouchscreen sensitivity of a touchscreen of the first device.
 17. Thecomputing system of claim 15, wherein the at least one device settingcomprises a font size for text displayed on a screen of the firstdevice.
 18. The computing system of claim 11, wherein the at least onecomputer-readable medium is encoded with additional instructions that,when executed by the at least one processor, further cause the computingsystem to: receive image data representing an image acquired by a cameraof the first device; determine that the image data corresponds to storedimage data associated with the profile; and determine that theindividual is the user based at least in part on the image datacorresponding to the stored image data.
 19. The computing system ofclaim 11, wherein the at least one computer-readable medium is encodedwith additional instructions that, when executed by the at least oneprocessor, further cause the computing system to: receive first deviceusage data representing a first interaction with the first device;determine that the first device usage data corresponds to stored usagecharacteristic data associated with the profile; and determining thatthe individual is the user based at least in part on the first deviceusage data corresponding to the usage characteristic data.
 20. Thecomputing system of claim 19, wherein the at least one computer-readablemedium is encoded with additional instructions that, when executed bythe at least one processor, further cause the computing system to: priorto receiving the first audio data, receive second audio datacorresponding to audio in the environment; determine that the secondaudio data corresponds to the previously stored voice data; receivesecond device usage data representing a second interaction with thefirst device; determine, based at least in part on second audio datacorresponding to the previously stored voice data, that the userperformed the second interaction; determine, based at least in part onthe second device usage data, the usage characteristic data, the usagecharacteristic data representing how the user interacts with the firstdevice; associate the usage characteristic data with the profile; andconfigure operation of the first device based at least in part on theusage characteristic data.